Evaluating Text Quality: Judging Output Texts Without a Clear Source

We consider how far two attributes of text quality commonly used in MT evaluation – intelligibility and fidelity – apply within NLG. While the former appears to transfer directly, the latter needs to be completely re-interpreted. We make a crucial distinction between the needs of symbolic authors and those of end-readers. We describe a form of textual feedback, based on a controlled language used for specifying software requirements that appears well suited to authors’ needs, and an approach for incrementally improving the fidelity of this feedback text to the content model.